In various computer systems, historical log data is maintained (for example, in log files) for use in system recovery, problem determination and/or system maintenance. Typically, these log files have a limited amount of capacity to retain the historical data. After the capacity is met, at least some of the data records are moved from the log files to external storage devices, such as direct access storage devices (DASD), thereby providing additional space in the log file for more data.
At some point, data within the log file and/or on the external storage devices is no longer needed. For example, there is no need to keep the data once it passes its retention requirements. The saving of data past its usefulness adversely affects system performance in a number of ways. For example, when unnecessary data is retained and the log file needs to be browsed to recover logged data during recovery of a failure, the browser may have to deal with potentially large amounts of unnecessary data, thereby slowing down the recovery process. Additionally, when the saving of unwanted data records requires the use of external storage devices, which typically provide slower access to the data, the reading of the data takes longer, thus, affecting system performance.
Therefore, it is beneficial to delete any unnecessary data from the log files. However, generally, there are a number of restrictions placed on what data can be deleted. In one example, data may only be deleted from the tail of the log file. Thus, if the data record at the tail is still necessary, then no data can be deleted from the log file. This forces the saving of many unnecessary or unwanted data records.
The above problem is exacerbated when a log file is shared by multiple systems and the data record at the tail is owned by a system that has failed. In that case, the data record may not be deleted for a very long time, if at all, causing the saving of an increasing number of unwanted data records.
Thus, a need exists for a technique that allows the removal of unnecessary data records from a log file, even when the data record at the tail is still wanted. A further need exists for a tail compression technique that is applicable to multisystem log streams.